Word2vec Skip-Gram Dimensionality Selection via Sequential Normalized Maximum Likelihood

In this paper, we propose a novel information criteria-based approach to select the dimensionality of word2vec Skip-gram (SG). From perspective probability theory, SG is considered as an implicit distribution estimation under assumption that there exists true contextual among words. Therefore, apply criteria with aim selecting best so corresponding model can be close possible distribution. We examine following for selection problem: Akaike’s Information Criterion (AIC), Bayesian (BIC), and Sequential Normalized Maximum Likelihood (SNML) criterion. SNML total codelength required sequential encoding data sequence on basis minimum description length. The proposed applied both original Negative Sampling clarify idea using criteria. Additionally, suffers from computational disadvantages, introduce heuristics its efficient computation. Moreover, empirically demonstrate outperforms BIC AIC. comparison other evaluation methods word embedding, selected by significantly closer optimal obtained analogy or similarity tasks.